Ultrasonic imaging device and image processing device

ABSTRACT

An object of the invention is to provide a user with information that serves as a material for determining whether an image generated by processing including a neural network is valid. A reception signal output by an ultrasonic probe that has received an ultrasonic wave from a subject is received, and an ultrasonic image is generated based on the reception signal. A trained neural network receives the reception signal or the ultrasonic image, and outputs an estimated reception signal or an estimated ultrasonic image. A validity information generation unit generates information indicating validity of the estimated reception signal or the estimated ultrasonic image by using one or more of the reception signal, the ultrasonic image, the estimated reception signal, the estimated ultrasonic image, and output of an intermediate layer of the neural network.

CROSS-REFERENCE TO RELATED APPLICATION

The present application claims priority from Japanese application JP2019-072291, filed on Apr. 4, 2019, the contents of which is herebyincorporated by reference into this application.

TECHNICAL FIELD

The present invention relates to an ultrasonic imaging technique forimaging an image in a subject by using an ultrasonic wave, and relatesto a technique of using an algorithm trained by a machine learningmethod for image reconstruction.

BACKGROUND ART

An ultrasonic imaging technique is a technique for non-invasivelyimaging an inside of a subject such as a human body using an ultrasonicwave (a sound wave that is not intended to be heard, generally a soundwave having a high frequency of 20 kHz or higher).

In recent years, due to advances in a machine learning techniquecentering on techniques such as a neural network and deep learning, aplurality of examples of reconstruction processing and image qualityimprovement processing, which use the machine learning technique, havebeen disclosed in imaging that is performed by using the ultrasonicimaging technique. By training the neural network using a set of inputdata to the neural network and target training data for output of theneural network, desired output can be obtained with high accuracy evenfor unknown input data. If a signal before being imaged is the inputdata and imaged data is the training data, the neural network performsimage reconstruction processing, and if the imaged data is used for boththe input data and the training data, the neural network can alsoimprove the image quality.

For example, Patent Literature 1 discloses an ultrasonic imaging systemthat outputs image data from a neural network, which uses an ultrasonicecho signal, a signal beam formed based on the echo signal, or both asinput to the neural network. The neural network is trained by themachine learning method and can be used to replace ultrasonic imagingprocessing in the related art, to obtain higher quality images, andobtain tissue property information, blood flow information, and the likewithout giving an explicit physical model.

CITATION LIST Patent Literature

PTL 1: WO 2018/127497

SUMMARY OF INVENTION Technical Problem

The neural network usually determines weights used for calculation ateach node through training based on a large amount of data, and canoutput a highly accurate target image, signal, and the like. Whenprocessing based on data is compared with model-based processing, whichis determined in advance, it may be difficult to intuitively understanda behavior of the neural network. In particular, it is difficult topredict the behavior of the neural network for unknown input. Therefore,it is difficult for a person who has viewed an image, a signal, or thelike output from the neural network to determine whether the output isvalid from only the output.

The ultrasonic imaging device described in PTL 1 has a configuration inwhich the neural network trained by machine learning is included in animaging process, and thus the neural network generates an image from areception signal that is unknown input. Therefore, it is difficult for auser who has viewed an image generated and displayed by the neuralnetwork to determine whether the image is a valid image.

An object of the invention is to provide a user with information thatserves as a material for determining whether an image generated byprocessing including a neural network is valid.

Solution to Problem

In order to achieve the above object, an ultrasonic imaging device ofthe invention includes: an image generation unit configured to receive areception signal output by an ultrasonic probe that has received anultrasonic wave from a subject, and generate an ultrasonic image basedon the reception signal; a trained neural network configured to receivethe reception signal or the ultrasonic image generated by the imagegeneration unit, and output an estimated reception signal or anestimated ultrasonic image; and a validity information generation unitconfigured to generate information indicating validity of the estimatedreception signal or the estimated ultrasonic image by using one or moreof the reception signal, the ultrasonic image, the estimated receptionsignal, the estimated ultrasonic image, and output of an intermediatelayer of the neural network.

Advantageous Effect

According to the invention, since the information indicating thevalidity of the image generated by using the neural network can bedisplayed by the ultrasonic imaging device, a user can determinate thevalidity of the image.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1A is a perspective view of an entire ultrasonic imaging deviceaccording to an embodiment, and FIG. 1B is a block diagram showing aschematic configuration of the ultrasonic imaging device of the presentembodiment.

FIG. 2 is a block diagram showing a configuration of the entireultrasonic imaging device according to the embodiment.

FIGS. 3A to 3C are block diagrams showing details of a data flow from areception signal processing unit to an image processing unit in anultrasonic imaging device main body according to the embodiment.

FIGS. 4A and 4B are graphs showing a predetermined relationship betweenan absolute value of a difference (feature value) and a value indicatingvalidity.

FIG. 5 is a conceptual diagram showing a calculation flow when a neuralnetwork is trained according to Embodiment 1.

FIG. 6 is a flowchart showing a flow of operations according toEmbodiment 1.

FIG. 7 is a diagram showing examples of ultrasonic images generated byan image processing unit according to Embodiment 1.

FIG. 8 is a flowchart showing a method for changing operations of theimage processing unit in accordance with input of a user according toEmbodiment 1.

FIG. 9 is a block diagram showing details of a configuration and a dataflow from a reception signal processing unit to an image processing unitin an ultrasonic imaging device main body according to Embodiment 2.

FIG. 10 is an explanatory diagram showing an example of a configurationof a neural network according to Embodiment 2.

FIG. 11 is a block diagram showing a data flow during changing ofprocessing of a validity information generation unit according to awavelength of an ultrasonic wave to be transmitted according toEmbodiment 3.

FIG. 12 is a conceptual diagram showing the data flow during thechanging of the processing of the validity information generation unitaccording to the wavelength of the ultrasonic wave to be transmittedaccording to Embodiment 3.

FIG. 13 is a block diagram showing a data flow when an image processingunit changes image processing based on validity information according toEmbodiment 4.

FIG. 14 is a block diagram showing the data flow when the imageprocessing unit changes the image processing based on the validityinformation according to Embodiment 4.

DESCRIPTION OF EMBODIMENTS

An embodiment of the invention will be described with reference to thedrawings.

As shown in FIGS. 1A and 1B, an ultrasonic imaging device according tothe present embodiment includes an image generation unit 108, a trainedneural network 109, and a validity information generation unit 110. Theimage generation unit 108 receives a reception signal output from anultrasonic probe 102 and generates an ultrasonic image based on thereception signal. The neural network 109 receives the reception signalor the ultrasonic image that is generated by the image generation unit108 and outputs an estimated reception signal or an estimated ultrasonicimage. The validity information generation unit 110 generatesinformation indicating validity of the estimated reception signal or theestimated ultrasonic image by using one or more of the reception signal,the ultrasonic image, the estimated reception signal, the estimatedultrasonic image, and output of an intermediate layer of the neuralnetwork 109.

For example, the validity information generation unit 110 is configuredto perform a calculation (for example, a calculation for obtaining adifference) of comparing two or more of the reception signal, theultrasonic image, the estimated reception signal, the estimatedultrasonic image, and the output of the intermediate layer of the neuralnetwork to generate the information indicating the validity based on acalculation result (difference).

As another example, the validity information generation unit 110 isconfigured to extract feature value from one of the reception signal,the ultrasonic image, the estimated reception signal, the estimatedultrasonic image, and the output of the intermediate layer of the neuralnetwork, and obtain a value indicating validity corresponding to theextracted feature value based on a predetermined relationship betweenthe feature value and the value indicating the validity.

Because of such a configuration, the ultrasonic imaging device of thepresent embodiment can display the estimated ultrasonic image or theultrasonic image generated by the image generation unit from theestimated reception signal, and the information indicating the validityof the estimated ultrasonic image or the ultrasonic image on the imagedisplay unit 104. Therefore, a user can easily determine the validity ofthe image and the like output from the neural network 109.

Embodiment 1

An ultrasonic imaging device 100 according to Embodiment 1 will bedescribed in detail with reference to FIGS. 1A, 2 and 3A.

In Embodiment 1, the validity information generation unit 110 obtains adifference between a reception signal or an ultrasonic image input tothe neural network 109 and an estimated reception signal or an estimatedultrasonic image output from the neural network 109, and generatesinformation indicating validity based on the difference.

FIG. 1A shows a perspective view of the entire ultrasonic imaging device100, FIG. 2 shows a schematic configuration of the entire ultrasonicimaging device 100, and FIG. 3A shows a detailed configuration diagramof a part of the device. The ultrasonic imaging device 100 includes anultrasonic imaging device main body 101, the ultrasonic probe 102including one or more ultrasonic elements 113 for transmitting andreceiving ultrasonic waves, a console 103 where a user inputs aparameter, and the image display unit 104 displaying the ultrasonicimage.

The ultrasonic imaging device main body 101 includes a transmission beamformer 106, a transmission and reception switch 107 switching betweentransmission and reception of signals between the ultrasonic probe 102and the main body 101, the image generation unit 108, the trained neuralnetwork 109, the validity information generation unit 110 generatinginformation 206 indicating validity of output of the neural network 109,an image processing unit 112, and a control unit 105 transmitting acontrol signal to each component of each of the above-describedcomponents 106 to 112 of the ultrasonic imaging device. The console 103is connected to the control unit 105 of the main body 101, and the imagedisplay unit 104 is connected to the image processing unit 112 to formthe ultrasonic imaging device 100.

The transmission beam former 106 generates a transmission signal delayedby a predetermined amount, and outputs the transmission signal to aplurality of the ultrasonic elements 113 forming the ultrasonic probe102. Accordingly, each of the plurality of ultrasonic elements 113transmits an ultrasonic wave delayed by the predetermined amount to asubject 114. The transmitted ultrasonic wave is reflected by the subject114, returned to the ultrasonic element 113, received by the ultrasonicelement 113, and converted into a reception signal. The reception signalis converted into a digital signal by an AD converter (not shown),becomes RF data 201, and is sent to the image generation unit 108 viathe transmission and reception switch 107.

The image generation unit 108 processes the RF data 201 and generates anultrasonic image 202 that is input data of the neural network 109. Theimage generation unit 108 performs general signal processing on RF datasuch as low-pass, high-pass, and band-pass filter processing, orperforms phasing processing using a delay-and-sum method or the like toreconstruct an ultrasonic image. The ultrasonic image 202 may be aso-called B-mode image, a Doppler image for viewing a flow, or anelastic information image for viewing hardness of a tissue. Further,processing performed by the image generation unit 108 may includevarious processing performed when generating these images.

The neural network 109 is a network that receives the ultrasonic image202 and outputs an estimated ultrasonic image (hereinafter referred toas an estimated image) 203, and is a trained network trained in advanceusing training data by a machine learning method. An example of thetraining data will be described later. The neural network 109 may be anynetwork trained by the machine learning method, and, for example, aconvolutional neural network or a recursive neural network can be used.

The image generation unit 108 generates an ultrasonic image of the sametype as the input data used by the neural network 109 for training. Theestimated image 203 output from the neural network 109 has the same dataformat as the ultrasonic data 202. That is, when the ultrasonic image202 is in a two-dimensional image format, the estimated image 203 isalso in a two-dimensional image format.

The validity information generation unit 110 generates the validityinformation 206 that is information indicating validity of the estimatedimage 203 output from the neural network 109. Here, the validityinformation generation unit 110 performs calculation (for example,calculation for obtaining a difference) using the estimated image 203and the ultrasonic image 202 to calculate the validity information 206.

The image generation unit 108, the neural network 109, and the validityinformation generation unit 110 can be implemented by software, and apart of or all the units can also be implemented by hardware. When theabove units are implemented by software, the units are configured with aprocessor such as a central processing unit (CPU) or a graphicsprocessing unit (GPU) and a memory, and implement functions of the imagegeneration unit 108, the neural network 109, and the validityinformation generation unit 110 by reading and executing programs storedin advance in the memory. Further, when the above units are implementedby hardware, a circuit may be designed using a custom IC such as anapplication specific integrated circuit (ASIC) or a programmable IC suchas a field-programmable gate array (FPGA) so as to implement at leastoperations of the image generation unit 108, the neural network 109, andthe validity information generation unit 110.

The operation of the validity information generation unit 110 will bedescribed. The validity information generation unit 110 calculates adifference between values of image elements corresponding to theultrasonic image 202 input to the neural network 109 and the estimatedimage 203 of the neural network, and generates an absolute valuethereof. The validity information generation unit 110 outputs theabsolute value of the difference as the validity information 206.

Further, the validity information generation unit 110 may obtain thedifference between the image element values corresponding to theultrasonic image 202 and the estimated image 203, further obtain theabsolute value thereof, and obtain a value indicating validitycorresponding to the obtained difference with reference to apredetermined relationship 205 between the absolute value of thedifference and a value indicating the validity. The predeterminedrelationship 205 between the absolute value of the difference and thevalue indicating the validity may be stored in a memory 111 as a table,or as shown in FIG. 4A, a graph or a function indicating therelationship 205 between the difference (feature value) and the valueindicating the validity (validity information) may be predetermined andstored in the memory 111, and the validity information generation unit110 may read out from the memory 111 and use the relationship 205. In anexample of FIG. 4A, it is set that if the difference is smaller than acertain threshold, the value indicating the validity is large, and ifthe difference is larger than the certain threshold, the valueindicating the validity is small. Further, the predeterminedrelationship 205 between the absolute value of the difference and thevalue indicating the validity may be set based on data used fortraining.

One image element here does not necessarily need to be one imageelement, and may be a region having a predetermined size. For example, aregion having a predetermined number of pixels can be set to one imageelement. In that case, as an pixel value of the image element, arepresentative value obtained by a predetermined calculation method suchas an average value, a maximum value, or a minimum value of the pixelsconstituting the image elements is used.

In the above description, the case where the ultrasonic image 202 andthe estimated image 203 have the same image size has been described, butthe ultrasonic image 202 and the estimated image 203 may have differentsizes and data formats. In that case, the validity informationgeneration unit 110 may calculate a value indicating validity for anycorresponding data included in the ultrasonic image 202 and theestimated image 203.

The calculation of the validity information 206 by the validityinformation generation unit 110 can take not only the method ofobtaining the difference shown here but also various calculations forcomparing the estimated image 203 and the ultrasonic image 202. Forexample, instead of the difference, a value obtained by normalizing adifference between two images with a maximum signal intensity in theimages and peak signal to noise ratio (PSNR) may be used, or acalculation such as a structure similarity (SSIM) index for comparingeach region in the images may be used.

For example, the validity information generation unit 110 may generatethe validity information 206 by calculating an image feature value ineach of the estimated image 203 and the ultrasonic image 202 for eachregion set in the image, and comparing the image feature values witheach other. The image feature value may be calculated using a textureanalysis method, for example, a feature value calculated by a textureanalysis using a co-occurrence matrix may be used.

The image processing unit 112 generates an ultrasonic image 207 to bedisplayed on the image display unit 104 based on the estimated image 203and the validity information 206. Accordingly, not only the estimatedimage 203 but also the validity information 206 as a basis fordetermining the validity of the estimated image 203 can be displayed tothe user. The image display unit 104 displays the ultrasonic image 207generated by the image processing unit 112 to the user. The ultrasonicimage 207 will be described in detail later.

Here, the trained neural network 109 will be described. The neuralnetwork 109 is trained in advance using the training data, whereby aweight for each node is determined. A training method of the neuralnetwork 109 will be described with reference to FIG. 5.

For training of the neural network 109, training input data 211 andtraining reference data 210 is used as the training data. The traininginput data 211 is data generated by the similar processing as that forgenerating the ultrasonic image 202. The training reference data 210 istarget data to be output from the neural network. Weights of nodesincluded in a plurality of layers constituting the neural network 109are optimized while inputting training input data 211 to the neuralnetwork 109 and referring to the training reference data 210.Specifically, training output data 212 that is the output of the neuralnetwork 109 when the training input data 211 is input and the trainingreference data 210 are compared by a loss function 213, and weightupdate 214 of the node of the neural network 109 is performed such thatthe loss function is minimized. For the weight update 214, for example,a back-propagation method is used.

For example, an image reconstructed using reception signals obtained bya small number of transmissions is used as the training input data 211,and an image reconstructed using reception signals respectively obtainedby a larger number of transmissions than that of the training input data211 is used as the training reference data 210. In other words, anultrasonic image in which a density of transmission scanning lines ishigher than that of the training input data 211 can be used as thetraining reference data 210. In this way, the neural network 109 afterbeing trained can output, from the ultrasonic image 202, the estimatedimage 203 that estimates the image reconstructed from the receptionsignals when the number of transmissions is large. The training data mayuse an ultrasonic image as input data and may use an ultrasonic image inwhich a density of at least one of the transmission scanning line and areception scanning line is higher than that of the input data 211 as thetraining reference data 210.

A reception signal (RF data) can be used as the training input data 211,and a reception signal, obtained by setting a frequency of atransmission signal of an ultrasonic wave to be transmitted to thesubject 114 to be higher than a transmission signal when the receptionsignal of the training input data is obtained, can be used as thetraining reference data 210. The neural network 109 after being trainedby such training data 211 and training reference data 210 can output,from the reception signal (RF data) 201, an estimated reception signal223 when the frequency of the transmission signal is set higher.Similarly, an ultrasonic image can be used as the training input data211, and an ultrasonic image, obtained by setting a frequency of atransmission signal of an ultrasonic wave to be transmitted to thesubject 114 to be higher than a transmission signal when an ultrasonicimage of the training input data is generated, can be used as thetraining reference data 210. The neural network 109 after being trainedby such training input data 211 and training reference data 210 canoutput, from the ultrasonic image 202, the estimated image 203 forestimating the ultrasonic image when the frequency of the transmissionsignal is set higher.

The training of the neural network 109 may be performed using thetraining input data 211 and the training reference data 210 imaged bythe same ultrasonic imaging device 100. Further, after training ofanother neural network having the same structure is performed by usingthe training input data and the training reference data 210 imaged usingdifferent devices, only the weights are stored in the neural network 109of the ultrasonic imaging device 100.

The relationship 205 between the difference (feature value) used by thevalidity information generation unit 110 to generate the validityinformation and the value indicating the validity may be set based on abehavior of the neural network 109 with respect to the training inputdata 211. For example, the training input data 211 is input as theultrasonic image 202 to the trained neural network 109 to generate theestimated image 203, and the difference (feature value) between theestimated image 203 and the ultrasonic image 202 is calculated. Thecalculation is performed for each of a plurality of pieces of thetraining input data 211, and a probability distribution of a pluralityof the obtained differences is calculated as shown in FIG. 4B. Therelationship 205 between the difference (feature value) and the valueindicating the validity is set as shown in FIG. 4B such that the largerthe obtained probability distribution is, the larger the valueindicating the validity (validity information) is.

Specifically, as shown in FIG. 4B, when the training input data 211 isinput to the neural network 109, the difference between the estimatedimage 203 output from the neural network 109 and the training input data211 (ultrasonic image 202) is obtained, and the probability distributionof the difference (a hatched histogram in FIG. 4B) is obtained. Therelationship 205 (shown by a dotted line in FIG. 4B) between thedifference (feature value) and the validity information is generatedsuch that the probability of the obtained difference is proportional tothe value indicating the validity (validity information).

In other words, within a range (range 411 in FIG. 4B) of a distributionof the difference between a plurality of pieces of the training inputdata 211 and a plurality of pieces of the output data 203 outputrespectively when the plurality of pieces of training input data 211used for training (learning) of the neural network 109 is input to thetrained neural network 109, the value indicating the validity is set toa larger value than other ranges. Accordingly, when the behavior (theobtained estimated image 203) when the ultrasonic image 202 is input tothe neural network 109 is similar as the behavior when the traininginput data 211 is input, the validity information generation unit 110can output a large value indicating the validity based on FIG. 4B.

The relationship 205 between the difference (feature value) and thevalue indicating the validity may be determined regardless of thetraining input data 211. For example, as already described withreference to FIG. 4A, when the difference is equal to or smaller than acertain threshold, as valid output, a large value may be output as avalue (information) indicating validity, and when the difference issmaller than the threshold, as invalid output, a small value may beoutput as a value (information) indicating validity.

Next, operations of the ultrasonic imaging device of the presentembodiment will be described in order with reference to FIG. 6.

First, in step S101, the transmission beam former 106 transmits atransmission signal to the ultrasonic element 113 of the probe 102. Theultrasonic element 113 transmits an ultrasonic wave to the subject 114.The ultrasonic element 113 receives an ultrasonic wave interacting withthe subject 114, and the image generation unit 108 performs the signalprocessing, the phasing processing, and the like on the reception signal(RF data 201) to generate the ultrasonic image 202 (step S102). Next,the neural network 109 receives the ultrasonic image 202 and outputs theestimated image 203 (step S103).

Next, the validity information generation unit 110 obtains thedifference between the estimated image 203 and the ultrasonic image 202and sets the absolute value as validity information, or based on thedifference, generates the value indicating the validity (validityinformation) 206 with reference to the predetermined relationship 205(step S104). Next, the image processing unit 112 generates theultrasonic image 207 based on the estimated image 203 and the validityinformation 206 (step S105). The image display unit 104 displays theultrasonic image 207 (step S106).

Based on input received from the user by the console 103, it isdetermined whether the imaging is to be ended. If the imaging is not tobe ended, the processing returns to step S101, the same operation isrepeated, and the ultrasonic image 207 displayed on the image displayunit 104 in step S106 is updated. If the imaging is ended, a series ofoperations are ended (step S107).

With the above procedure, the user can determine the validity of theestimated image 203 output from the neural network 109 by viewing theultrasonic image 207 displayed on the image display unit 104.

Examples of images generated by the image processing unit 112 anddisplayed on the image display unit 104 will be described with referenceto FIG. 7.

As shown at (a) in FIG. 7, the image processing unit 112 can generatethe ultrasonic image 207 in which the estimated image 203 and thevalidity information 206 are superimposed. In a superimposing method,black and white may be used for luminance of the estimated image 203 anda color such as red may be used for the validity information 206.

Further, as shown at (b) in FIG. 7, the image processing unit 112 maygenerate the ultrasonic image 207 in which the estimated image 203 andthe validity information 206 are arranged adjacently. Accordingly, theuser can compare the estimated image 203 with the validity information206.

Further, as shown at (c) in FIG. 7, the image processing unit 112 maygenerate the ultrasonic image 207 in which the validity information 206is displayed in a display region provided in the estimated image 203. Bythese methods, the user can know the validity information 206 whileobserving the estimated image 203.

The image processing unit 112 may generate the ultrasonic image 207 inwhich the validity information 206 is intermittently displayed while theestimated image 203 is displayed. That is, the ultrasonic image 207 isgenerated in which the following is repeated at regular intervals: in acertain frame, the validity information 206 is superimposed on theestimated image 203, and in a certain frame, only the estimated image203 is set as the ultrasonic image 207. By this method, the user canobserve the estimated image 203 without being disturbed by the validityinformation 206, and can check the validity information 206 at regulartime intervals.

The image processing unit 112 may generate the ultrasonic image 207 inwhich the estimated image 203 is corrected based on the validityinformation 206. The ultrasonic image 207 may be generated by performingprocessing of, for example, reducing the luminance of the estimatedimage 203 for a region where the validity information 206 has a smallvalue, that is, a region where the validity is estimated to be low, andincreasing the luminance of the estimated image 203 for an area wherethe validity is estimated to be high. By such a method, the user is morelikely to focus on only image regions that are more valid.

The image processing unit 112 may take statistics for all regions or apart of the regions of the validity information 206, summarize values(information) indicating the validity of the regions into a certainnumerical value, and generate the ultrasonic image 207 in which thenumerical value is displayed on the ultrasonic image 207. Accordingly,the user can easily determine the validity of the estimated image 203 inthe region of interest by viewing only the numerical value.

Another example of the operations of the ultrasonic imaging device 100will be described with reference to FIG. 8. In the example of FIG. 8, inaccordance with input of the user, the ultrasonic imaging device 100displays validity information only when the user requests to displaydetermination material related to validity.

Steps S111 to S113 and steps S118 to S119 in FIG. 8 are similaroperations as steps S101 to S107 described in FIG. 6, and descriptionthereof will be omitted.

In step S114 of FIG. 8, whether to display the validity information isswitched based on information input by the user from the console 103.That is, when the user requests to display the validity information,operations similar to those in steps S104 to S105 in FIG. 6 areperformed in steps S116 to S117, the validity information generationunit 110 generates the validity information 206, and the imageprocessing unit 112 generates the ultrasonic image 207 from the validityinformation 206 and the estimated image 203.

On the other hand, if the user does not request to display the validityinformation in step S114, the image processing unit 112 performs imageprocessing based only on the estimated image 203 and generates theultrasonic image 207 (step S115).

In an operation example of FIG. 8, the ultrasonic imaging device 100 canprovide the validity information 206 only when the user needs thedetermination material related to the validity.

The input received by the console 103 from the user may include not onlypresence or absence of the display of the validity information as shownin FIG. 8 but also a setting of various parameters related to a methodof presenting the validity information. For example, the console 103 mayreceive, from the user, a setting of a display luminance gain of thevalidity information 206, a setting of a frame extinction rate at whichthe validity information 206 is intermittently displayed, a setting ofluminance reduction of the estimated image 203 to be reduced based onthe validity information 206 and the like.

Further, the user may switch a method of reflecting the validityinformation 206 onto the ultrasonic image 207. For example, the user mayselect one of the reflection methods (a)-(c) shown in FIG. 7.Accordingly, the determination material related to the validity of theestimated image 203 can be provided according to the interest of theuser related to the validity of the estimated image.

Modification of Embodiment 1

In Embodiment 1 described above, the neural network 109 outputs theestimated image 203 using the ultrasonic image 202 as the input data,but the present embodiment is not limited thereto. The input of theneural network 109 may use data at a time point such as when anultrasonic image is generated from the RF data 201. For example, thereceived RF data itself, RF data after reception beam forming (phasing),or data obtained by adding a plurality of pieces of the RF data afterreception beam forming may be used as the input data for the neuralnetwork 109. Further, ultrasonic image data after image processing,logarithmic compression and the like may be used as the input data ofthe neural network 109.

When the RF data 201 is used as the input data of the neural network109, the RF data in the same dimension can be output from the neuralnetwork 109 as the estimated signal 223.

As a specific example, as shown in FIG. 3B, the neural network 109 isconfigured to receive a reception signal 222 that is phased by thereception beam former 120 and output the estimated reception signal 223in the same dimension. The image generation unit 108 is arranged at asubsequent stage of the neural network 109, and the image generationunit 108 generates an ultrasonic image based on the estimated receptionsignal 223. The validity information generation unit 110 obtains adifference between the reception signal 222 input to the neural network109 and the estimated reception signal 223 output from the neuralnetwork 109, and calculates information indicating validity based on thedifference.

In a case of the configuration of FIG. 3B, for example, a receptionsignal can be used as input data for training the trained neural network109, and a reception signal obtained by setting a frequency of atransmission signal of an ultrasonic wave to be transmitted to thesubject 114 to be higher than a transmission signal when a receptionsignal of training input data is obtained can be used as trainingreference data. Accordingly, when the reception signal 222 is received,the trained neural network 109 can output the estimated reception signal223 obtained by setting the frequency of the transmission signal of theultrasonic wave to be transmitted to the subject 114 to be higher thanthe transmission signal when the reception signal 222 is obtained.

Further, the neural network 109 may be configured to use the RF data 201as input data and output an ultrasonic image as the estimated image 203.

As a specific example, as shown in FIG. 3C, the neural network 109 isconfigured to receive the reception signal 222 and output the estimatedultrasonic image 203. In this case, the image generation unit 108 isarranged in parallel with the neural network 109 at a subsequent stageof a reception beam former 120, and the image generation unit 108 isconfigured to generate an ultrasonic image from the reception signal222. The validity information generation unit 110 obtains a differencebetween an ultrasonic image generated by the image generation unit 108from the reception signal 222 and the estimated ultrasonic image 203output from the neural network 109, and calculates informationindicating validity based on the difference.

Embodiment 2

An ultrasonic imaging device according to Embodiment 2 will bedescribed. In the ultrasonic imaging device according to Embodiment 2,the validity information generation unit 110 extracts a feature valuefrom one of the reception signal (RF data), the ultrasonic image, theestimated reception signal, the estimated ultrasonic image, and theoutput of the intermediate layer of the neural network, and obtains avalue indicating validity corresponding to the extracted feature valuebased on the predetermined relationship 205 between the feature valueand the value indicating the validity.

FIG. 9 shows an example of a configuration of the ultrasonic imagingdevice according to Embodiment 2. In the configuration of FIG. 9, thevalidity information generation unit 110 generates the validityinformation 206 based on intermediate layer output 215 that is output ofan intermediate layer 115 of the neural network 109. This configurationwill be described with a focus on differences from Embodiment 1.

The neural network 109 is configured with a plurality of continuousfunctions, and each of the functions is called a layer. The intermediatelayer 115 refers to a function in the neural network 109, and an outputvalue of the function is referred to as the intermediate layer output215. For example, in a case of a forward propagation neural network, anoutput value of an activation function corresponds to the intermediatelayer output 215.

As shown in FIG. 9, the validity information generation unit 110receives the intermediate layer output 215. The validity informationgeneration unit 110 generates the validity information 206 based on theintermediate layer output 215. For example, the validity informationgeneration unit 110 reads the relationship 205 between the feature valueand the validity information, which is stored in the memory 111 inadvance, and generates the validity information 206 from theintermediate layer output 215 based on the relationship 205.

The relationship 205 between the feature value and the validityinformation, which is stored in the memory 111 in advance, is generated,for example, as follows. First, the training input data 211 is input tothe trained neural network 109, and the intermediate layer output 215 atthat time is obtained. The above processing is performed for theplurality of pieces of training input data 211 to obtain a plurality ofpieces of intermediate layer output 215. Information related to apattern (feature value) of the intermediate layer output 215 when thetraining input data 211 is input to the neural network 109 is obtainedby averaging the plurality of pieces of obtained intermediate layeroutput 215 or performing pattern classification.

Further, the relationship 205 between the feature value and the validityinformation is generated in advance such that the intermediate layeroutput 215 indicating output of a pattern (feature value) similar to acase where the training input data 211 is input has high validity, andthe intermediate layer output 215 indicating output of a pattern(feature value) different from the case where the training input data211 is input has low validity. The generated relationship 205 betweenthe feature value and the validity information is stored in the memory111.

Further, when the actual ultrasonic image 202 is input to the neuralnetwork 109, the validity information generation unit 110 receives theintermediate layer output 215 of the intermediate layer 115, and obtainsvalidity information (a value indicating validity) corresponding to theintermediate layer output 215 with reference to the relationship 205between the feature value and the validity information, which is readfrom the memory 111. When the value indicating the validity is large,the validity information generation unit 110 can determine that theultrasonic image 202 that is input data to the neural network 109 hasthe same behavior as the training input data 211. Therefore, thevalidity information generation unit 110 can determine whether theultrasonic image 202 is included in a range trained by the traininginput data 211 based on the value of the validity information, and ifthe ultrasonic image 202 is included in the range trained by thetraining input data 211, the validity information generation unit 110can output validity information indicating that the validity of theestimated image 203 to be output is high.

Further, in FIG. 9, a middle layer of the neural network 109 is used asthe intermediate layer 115, but the intermediate layer 115 is notlimited to the middle layer, and any layer between an input layer and anoutput layer may be used as the intermediate layer 115.

As shown in FIG. 10, when the neural network 109 in which the number ofnodes varies depending on a layer, the layer having the smallest numberof nodes may be used as the intermediate layer 115 and an output thereofmay be set as the intermediate layer output 215. Since it is generallysaid that the layer with the smallest number of nodes is easy to showfeatures, by using data output from the layer as the intermediate layeroutput 215, the validity information generation unit 110 can more easilydetermine whether the ultrasonic image 202 that is input data has thesame behavior as the training input data 211.

As input to the validity information generation unit 110, rather thanthe intermediate layer output 215, the ultrasonic data 202 or any dataduring processing in the image generation unit 108 may be used, or theestimated image 203 may be used, or any combination thereof may be used.By using the method, the validity information 206 of the estimated image203 can be generated even when the ultrasonic data 202 is in an RF dataformat and the estimated image 203 is in a different data format, suchas an image data format. That is, the validity information 206 of theestimated image 203 can be generated even when data formats of the inputand the output of the neural network 109 are different.

In FIG. 9, a configuration corresponding to FIG. 3A in which theultrasonic image 202 is input to the neural network 109 is shown as theconfiguration of the ultrasonic imaging device, but Embodiment 2 is notlimited thereto. Of course, other configurations such as a configurationin which a reception signal is input to the neural network 109 shown inFIGS. 3B and 3C and the like are possible.

Embodiment 3

An ultrasonic imaging device according to Embodiment 3 will be describedwith reference to FIGS. 11 and 12.

The ultrasonic imaging device according to Embodiment 3 has the similarconfiguration as that of Embodiment 1, but the validity informationgeneration unit 110 is different from that of Embodiment 1 in that animage element size for calculating validity information is changedaccording to a wavelength of a transmitted ultrasonic wave or a receivedultrasonic wave.

FIG. 11 is a diagram showing a configuration of a main part of theultrasonic imaging device according to Embodiment 3, and FIG. 12 is adiagram showing the image element size and the wavelength of thetransmission ultrasonic wave by which the validity informationgeneration unit 110 calculates the validity information.

The user sets, through the console 103, a wavelength of an ultrasonicwave 116 transmitted by the ultrasonic probe 102. The setting of thetransmission ultrasonic wave may be configured so that the wavelengthand a frequency of the ultrasonic wave can be directly set as numericalvalues, or may be switched indirectly by setting an imaging mode.

The control unit 105 instructs, according to a wavelength of thetransmission ultrasonic wave 116 set through the console 103, thetransmission beam former 106 to generate a transmission signal fortransmitting an ultrasonic wave having the wavelength. The transmissionbeam former 106 transmits a transmission signal to the ultrasonic probe102 via the transmission and reception switch 107. Accordingly, thetransmission ultrasonic wave 116 having the set wavelength istransmitted from the ultrasonic probe 102.

The control unit 105 sends information of a transmission wavelength tothe validity information generation unit 110, and the validityinformation generation unit 110 changes a size of a coordinate grid 301of the validity information 206 to be generated, according to theinformation of the transmission wavelength. FIG. 11 shows a case wherethe estimated image 203, operation data 204 of the intermediate layer115 of the neural network, and the ultrasonic data 202 are input to thevalidity information generation unit 110, but as described inEmbodiments 1 and 2, any combination of these data may be input to thevalidity information generation unit 110.

A wavelength of the reception signal 201 is detected in the imagegeneration unit 108, and the validity information generation unit 110may change the image element size according to the wavelength size.

A method for changing a size of the coordinate grid 301 will bedescribed in detail with reference to FIG. 12. Here, a case where thevalidity information 206 generated by the validity informationgeneration unit 110 is in a two-dimensional image format will bedescribed as an example. The coordinate grid 301 of the validityinformation 206 to be generated is configured with an x coordinate and ay coordinate. A region defined by the coordinate grid 301 is one imageelement. The validity information generation unit 110 changes an imageelement size of the validity information in an x direction 302 and animage element size of the validity information in a y direction 303according to a wavelength 305 of a waveform 304 of the transmissionultrasonic wave.

The validity information generation unit 110 may be configured to setthe image element size in the x direction 302 and the image element sizein the y direction 303 to a size proportional to the wavelength 305 ofthe transmission ultrasonic wave by a certain constant, or may be in aform in which the image element sizes 302 and 303 corresponding to thewavelength 305 are changed according to a predefined table.

The image element size in the x direction 302 and the image element sizein the y direction 303 may be the same, or may use different values.

By setting the coordinate grid 301 of the validity information in thisway, an appropriate grid size (image element size) can be set duringgeneration of the validity information 206, and a calculation cost canbe reduced.

The format of the validity information 206 is not limited to thetwo-dimensional image format, and may be a three-dimensional volumeformat, or further may be a three-dimensional or four-dimensional formathaving a plurality of frames. Further, for example, when a sector probeor a convex probe or the like is used as the ultrasonic probe, thecoordinate grid may be generated using another spatial coordinate systemsuch as a polar coordinate system instead of a Cartesian coordinatesystem.

Embodiment 4

An ultrasonic imaging device according to Embodiment 4 will be describedwith reference to FIGS. 13 and 14.

The ultrasonic imaging device of Embodiment 4 has a similarconfiguration as that of Embodiment 1, but an aspect of the ultrasonicimage 207 that is generated by the image processing unit 112 anddisplayed on the image display unit 104 is different from that ofEmbodiment 1.

In Embodiment 4, the image processing unit 112 performs processing ofgenerating the ultrasonic image 207 by using the validity information206 to update the estimated image 203, or performs processing such asselecting an image used to generate the ultrasonic image 207 from aplurality of the estimated images 203.

The processing in which the image processing unit 112 updates theestimated image 203 from the validity information 206 to generate theultrasonic image 207 will be described with reference to FIG. 13.

The image processing unit 112 generates the ultrasonic image 207 basedon the ultrasonic image 202, the validity information 206, and theestimated image 203. At this time, in accordance with the validityinformation 206, processing of changing how the estimated image 203 andthe ultrasonic image 202 are reflected on the ultrasonic image 207 isperformed.

For example, the image processing unit 112 assigns the estimated image203 to the ultrasonic image 207 for a region where the validityinformation 206 exceeds a certain threshold, that is, has high validity,and assigns the ultrasonic image 202 to the ultrasonic image 207 for aregion where the validity information 206 does not exceed the certainthreshold so as to generate the ultrasonic image 207.

Alternatively, the image processing unit 112 generates the ultrasonicimage 207 by adding the ultrasonic image 202 and the estimated image 203with a certain weight, and changes the weight at that time according tothe validity information 206. That is, when the validity information 206has a large value, the image processing unit 112 gives a high weight tothe estimated image 203, and when the validity information 206 has a lowvalue, the image processing unit 112 gives a high weight to theultrasonic image 202.

Further, the image processing unit 112 may generate the ultrasonic image207 by combining the method of switching assignment according to thevalidity information 206 and the method of changing the weight accordingto the validity information 206.

The image processing unit 112 causes the image display unit 104 todisplay the ultrasonic image 207 generated by such a method, and thusthe user can determine whether an image is valid by viewing theultrasonic image 207.

Further, the image processing unit 112 may be configured to display analert superimposed on the ultrasonic image 207 or output a sound inorder to notify an operator when the validity information satisfies acertain condition. Accordingly, when the validity information indicatesthat the validity is low, the user can be alerted not to use theestimated image 203.

Further, referring to FIG. 14, processing of the image processing unit112 when the neural network 109 of the ultrasonic processing deviceincludes a plurality of neural networks, each of the neural networksgenerates the estimated image 203, and the validity informationgeneration unit 110 generates a plurality of pieces of the validityinformation 206 will be descried. The validity information generationunit 110 generates each of the plurality of pieces of validityinformation 206 using the estimated image 203, the ultrasonic image 202,the intermediate layer output 215, or the like for each of the pluralityof neural networks.

In this case, in order to select or generate the ultrasonic image 207having higher validity based on the plurality of pieces of validityinformation 206, the image processing unit 112 is configured to assignthe estimated image 203 corresponding to the validity information 206having high validity among the plurality of pieces of validityinformation 206 to the ultrasonic image 207, and generate the ultrasonicimage 207.

Alternatively, the image processing unit 112 may be configured togenerate the ultrasonic image 207 by adding a plurality of the estimatedimages 203 with weights according to the validity information 206.

Further, the ultrasonic image 207 may be generated by combining themethod of switching assignment according to the validity information 206and the method of changing the weight according to the validityinformation 206.

The image processing unit 112 causes the image display unit 104 todisplay the ultrasonic image 207 generated by such a method, and thusthe user can determine whether an image is valid by viewing theultrasonic image 207.

In each of the embodiments described above, the user is a person whouses the ultrasonic imaging device. The user may be a doctor or asonographer. Further, the user may be an engineer such as a developer ora person who manages production during manufacturing. The invention maybe used by such users in development and manufacturing process forperformance verification during development and quality assurance duringmanufacturing. For example, the validity information 206 when a standardspecification phantom is imaged may be used as an index for performanceverification during development or as a part of determination materialfor quality assurance. Accordingly, efficient development, test, andmanufacturing can be performed.

REFERENCE SIGN LIST

-   100: ultrasonic imaging device-   101: ultrasonic imaging device main body-   102: ultrasonic probe-   103: console-   104: image display unit-   105: control unit-   106: transmission beam former-   107: transmission and reception switch-   108: image generation unit-   109: neural network-   110: validity information generation unit-   111: memory-   112: image processing unit-   113: ultrasonic element-   114: subject-   115: intermediate layer of neural network-   116: transmission ultrasonic wave-   201: RF data-   202: ultrasonic data-   203: estimated image-   205: relationship between feature value and validity information-   206: validity information-   207: ultrasonic image-   210: training reference data-   211: training input data-   212: training output data-   213: loss function-   214: weight update-   215: intermediate layer output

1. An ultrasonic imaging device, comprising: an image generation unitconfigured to receive a reception signal output by an ultrasonic probethat has received an ultrasonic wave from a subject, and generate anultrasonic image based on the reception signal; a trained neural networkconfigured to receive the reception signal or the ultrasonic imagegenerated by the image generation unit, and output an estimatedreception signal or an estimated ultrasonic image; and a validityinformation generation unit configured to generate informationindicating validity of the estimated reception signal or the estimatedultrasonic image by using one or more of the reception signal, theultrasonic image, the estimated reception signal, the estimatedultrasonic image, and output of an intermediate layer of the neuralnetwork.
 2. The ultrasonic imaging device according to claim 1, whereinthe validity information generation unit is configured to perform acalculation of comparing two or more of the reception signal, theultrasonic image, the estimated reception signal, the estimatedultrasonic image, and the output of the intermediate layer of the neuralnetwork to generate the information indicating the validity based on thecalculation result.
 3. The ultrasonic imaging device according to claim1, wherein the validity information generation unit is configured toobtain a difference between the reception signal or the ultrasonic imageinput to the neural network and the estimated reception signal or theestimated ultrasonic image output from the neural network, and generatethe information indicating the validity based on the difference.
 4. Theultrasonic imaging device according to claim 3, wherein the validityinformation generation unit is configured to obtain a value indicatingthe validity corresponding to the obtained difference with reference toa predetermined relationship between a difference and a value indicatingvalidity.
 5. The ultrasonic imaging device according to claim 4, whereinthe predetermined relationship between the difference and the valueindicating the validity is set such that when the difference is within apredetermined range, the corresponding value indicating the validity ishigher than in other ranges, and wherein the predetermined range is arange of a value distribution of differences between a plurality ofpieces of training input data used during training of the neural networkand a plurality of pieces of output data output respectively when theplurality of pieces of training input data used during training of theneural network is input to the trained neural network.
 6. The ultrasonicimaging device according to claim 1, wherein the neural network isconfigured to receive the ultrasonic image generated by the imagegeneration unit and output the estimated ultrasonic image, and whereinthe validity information generation unit is configured to obtain adifference between the ultrasonic image input to the neural network andthe estimated ultrasonic image output from the neural network, andcalculate the information indicating the validity based on thedifference.
 7. The ultrasonic imaging device according to claim 1,wherein the neural network is configured to receive the reception signaland output the estimated ultrasonic image, and wherein the validityinformation generation unit is configured to obtain a difference betweenthe ultrasonic image generated by the image generation unit from thereception signal and the estimated ultrasonic image output from theneural network, and calculate the information indicating the validitybased on the difference.
 8. The ultrasonic imaging device according toclaim 1, wherein the neural network is configured to receive thereception signal and output the estimated reception signal, wherein thevalidity information generation unit is configured to obtain adifference between the reception signal input to the neural network andthe estimated reception signal output from the neural network, andcalculate the information indicating the validity based on thedifference, and wherein the image generation unit is configured togenerate the ultrasonic image based on the estimated reception signal.9. The ultrasonic imaging device according to claim 1, wherein thevalidity information generation unit is configured to extract a featurevalue from one of the reception signal, the ultrasonic image, theestimated reception signal, the estimated ultrasonic image, and theoutput of the intermediate layer of the neural network, and obtain avalue indicating validity corresponding to the extracted feature valuebased on a predetermined relationship between the feature value and thevalue indicating the validity.
 10. The ultrasonic imaging deviceaccording to claim 1, wherein the trained neural network has trainedusing training data, and for the training data, an ultrasonic image isused as input data, and an ultrasonic image in which a density of atleast one of a transmission scanning line and a reception scanning lineis higher than that of the input data is used as training referencedata.
 11. The ultrasonic imaging device according to claim 1, whereinthe trained neural network has trained using training data, and for thetraining data, a reception signal that is output by the ultrasonic probewhen a transmission signal of an ultrasonic wave is transmitted from theultrasonic probe to the subject and the ultrasonic probe receives anultrasonic wave from the subject, is used as input data, and thereception signal output by the ultrasonic probe when a frequency of thetransmission signal transmitted to the subject is higher than that ofthe input data is used as training reference data.
 12. The ultrasonicimaging device according to claim 1, wherein the trained neural networkhas trained using training data, and for the training data, anultrasonic image or a reception signal generated from a reception signalthat is output by the ultrasonic probe when a transmission signal of anultrasonic wave is transmitted from the ultrasonic probe to the subjectand the ultrasonic probe receives an ultrasonic wave from the subject isused as input data, and the ultrasonic image generated from thereception signal output by the ultrasonic probe when a frequency of thetransmission signal transmitted to the subject is higher than that ofthe input data is used as training reference data.
 13. The ultrasonicimaging device according to claim 1, wherein the reception signal isobtained such that an ultrasonic wave is transmitted from the ultrasonicprobe to the subject, and an ultrasonic wave reflected by the subjectwhich is received by the ultrasonic probe, and wherein the validityinformation generation unit is configured to generate two-dimensional orthree-dimensional validity information in which a value indicating thevalidity is given for each image element arranged two-dimensionally orthree-dimensionally, and change a size of the image element according toa wavelength of the ultrasonic wave transmitted from the ultrasonicprobe or received by the ultrasonic probe.
 14. The ultrasonic imagingdevice according to claim 1, further comprising: a console configuredfor an operator to input information; and an image processing unitconfigured to generate an image in which the information indicating thevalidity is reflected on an ultrasonic image generated from theestimated ultrasonic image or the estimated reception signal generatedby the neural network, wherein the image processing unit is configuredto change a method of reflection in a process of reflecting theinformation indicating the validity on the ultrasonic image generatedfrom the estimated ultrasonic image or the estimated reception signalaccording to information input to the console.
 15. The ultrasonicimaging device according to claim 1, further comprising: an imageprocessing unit configured to warn a user when the validity indicated bythe information indicating the validity is lower than a predeterminedcondition.
 16. The ultrasonic imaging device according to claim 1,further comprising: an image processing unit, wherein the trained neuralnetwork includes a plurality of neural networks, and each of the neuralnetworks generates the estimated reception signal or the estimatedimage, wherein the validity information generation unit is configured togenerate the information indicating the validity for each of a pluralityof the estimated images generated by the plurality of neural networks,and wherein the image processing unit is configured to select orgenerate more valid information from a plurality of pieces of theinformation indicating the validity.
 17. An image processing device,comprising: a trained neural network configured to receive a receptionsignal of an ultrasonic wave or an ultrasonic image, and output anestimated reception signal or an estimated ultrasonic image; and avalidity information generation unit configured to generate informationindicating validity of the estimated reception signal or the estimatedultrasonic image by using one or more of the reception signal, theultrasonic image, the estimated reception signal, the estimatedultrasonic image, and output of an intermediate layer of the neuralnetwork.